Apparatus for and method of extracting image

ABSTRACT

The luminance signal of a reference image obtained by an infrared camera is adjusted such that the luminance signal of the reference image obtained is greater at all times than the luminance signal of a searching image obtained by an infrared camera. Correlative calculations are performed on the reference image and the searching image to detect one object in the reference image and the searching image without detecting the background of the images in error.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an apparatus for and a method ofextracting an image of one object by comparing obtained images of theobject, the images by a plurality of imaging units.

2. Description of the Related Art

There has been developed a vehicle vicinity monitoring apparatus forobtaining images of one object with two imaging units mounted on avehicle, for example, either measuring the distance up to the objectbased on the parallax between the obtained images, or measuring theposition in an actual space of the object with respect to the vehicle,and informing the driver of whether there is an obstacle ahead of thevehicle or not (see Japanese Laid-Open Patent Publication No.2003-216937).

For extracting the same object from the two images obtained by the twoimaging units, correlative calculations are used to determine the sum ofabsolute differences (SAD) between image data of an object areaidentified in one of the images and image data of the other image, andto regard an area where the sum is minimum as an object area in theother image.

The imaging units, each comprising a CCD camera or the like, may notnecessarily have identical video characteristics because they tend tohave different optical characteristics and different photoelectricconversion characteristics of respective pixels and photoelectricallyconverted video signals tend to contain noise. If the above correlativecalculations are performed on image signals from the two imaging unitswhose video characteristics are different from each other, then thebackground of the object in question may possibly be recognized inerror, and different objects may possibly be judged as the same object.

SUMMARY OF THE INVENTION

It is a general object of the present invention to provide an apparatusfor and a method of reliably extracting an image of one object fromimages obtained by a plurality of imaging units even if the imagingunits have different characteristics.

A primary object of the present invention is to provide an apparatus forand a method of extracting an image of a desired object withoutrecognizing a background image in error.

According to an aspect of the present invention, the luminance signal ofa reference image obtained by a reference imaging unit is adjusted suchthat the luminance signal of the reference image is greater by apredetermined amount than the luminance signal of a searching imageobtained by a searching imaging unit. Then, a reference image of aparticular object area set in the reference image is compared with thesearching image. An image corresponding to a particular object in thereference image can reliably be searched for from the searching imagewithout detecting the background of the reference image or the searchingimage in error.

According to another aspect of the present invention, an image having agreatest luminance signal is selected from images obtained by aplurality of imaging units, and an image of a particular object area setin the selected image is compared with an image from another of theimaging units. An image of a particular object can reliably be extractedfrom the image of the other imaging unit.

The above and other objects, features, and advantages of the presentinvention will become more apparent from the following description whentaken in conjunction with the accompanying drawings in which preferredembodiments of the present invention are shown by way of illustrativeexample.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic perspective view of a night vision systeminstalled on a vehicle which incorporates an image extracting apparatusaccording to an embodiment of the present invention;

FIG. 2 is a functional block diagram of an ECU of the night visionsystem shown in FIG. 1;

FIG. 3 is a block diagram of a normal mode execution unit in the ECUshown in FIG. 2;

FIG. 4 is a perspective view of an aiming target control apparatusinstalled in a manufacturing plant;

FIG. 5 is a perspective view of a service aiming adjustment apparatusinstalled in a service factory or the like;

FIGS. 6 through 10 are flowcharts of an aiming process;

FIG. 11 is a flowchart of an object searching process in a normal mode;

FIG. 12 is a diagram illustrative of an object extracting process in thenormal mode and a luminance adjusting LUT (Look-Up Table) that is setfor extracting an object;

FIG. 13 is a diagram illustrative of a process of setting a luminanceadjusting LUT;

FIG. 14 is a diagram illustrative of a process of setting a luminanceadjusting LUT;

FIG. 15 is a diagram of a luminance adjusting LUT; and

FIG. 16 is a block diagram of a normal mode execution unit according toanother embodiment of present invention.

DESCRIPTION OF THE PREFFERED EMBODIMENTS

Apparatus for and methods of extracting an image according to preferredembodiments of the present invention will be described below withreference to the accompanying drawings.

As shown in FIG. 1, a night vision system (vehicle vicinity monitoringapparatus) 10 according to an embodiment of the present invention isinstalled on a vehicle 12. The night vision system 10 has an ECU(Electronic Control Unit) 14 serving as a main controller, a pair ofleft and right infrared cameras (imaging units) 16R, 16L, an HUD(Head-Up Display) 18 for displaying a detected image, a speaker 20 foroutputting an alarm sound, a speed sensor 22 for detecting a runningspeed, a yaw rate sensor 24 for detecting a yaw rate of the vehicle 12when the vehicle 12 is driven, a solar radiation sensor 26, a headlightswitch 28, a main switch 30 for selectively activating and inactivatingthe night vision system 10, and a connector 32 for connecting the nightvision system 10 to an external computer system. These components of thenight vision system 10 may be connected to each other by intravehicularcommunication lines that are used by other systems on the vehicle 12.

The infrared cameras 16R, 16L are mounted respectively in the right andleft ends of a horizontal grill hole defined in a lower bumper region.The infrared cameras 16R, 16L are oriented forwardly at respectivesymmetrical positions and horizontally spaced from each other by aninter-camera distance (also referred to as “base length”) B. Each of theinfrared cameras 16R, 16L detects far-infrared radiation to obtain aninfrared image in which higher-temperature areas represent higherluminance, and supplies the obtained image to the ECU 14.

The HUD 18 is disposed on an upper surface of an instrumental panel at aposition directly in front of the driver seated on a driver's seat ofthe vehicle 12, while trying not to obstruct the front vision of thedriver. When the night vision system 10 is turned off, the HUD 18 isretracted down in the instrumental panel. If it is judged that thepresent time is nighttime based on information from the solar radiationsensor 26 and also that the headlights (or fog lamps) are turned onbased on information from the headlight switch 28, then the HUD 18 popsup from the instrumental panel when the main switch 30 is turned on. TheHUD 18 has an image display panel comprising a concave mirror forreflecting and projecting an image sent from within the instrumentalpanel. The night vision system 10 may be automatically activated by anautomatic lighting function regardless of whether the main switch 30 isoperated or not. The luminance of the image display panel of the HUD 18may be made adjustable by a suitable switch.

The ECU 14 processes two infrared images obtained by the respectiveinfrared cameras 16R, 16L to detect heat-source objects based on theparallax between the infrared images, and displays the detectedheat-source objects as white silhouettes on the HUD 18. When the ECU 14identifies a pedestrian among the heat-source objects, the ECU 14controls the speaker 20 to output an alarm sound and also controls theHUD 18 to highlight the identified pedestrian with a surrounding framehaving a striking color for thereby drawing the driver's attention. TheECU 14 performs such an attention drawing function at such good timingto allow the driver to take a sufficient danger avoiding action, bypredicting a period of time until the vehicle 12 reaches the position ofthe pedestrian in a predetermined speed range.

In order for the infrared cameras 16R, 16L to be able to accuratelydetermine the positions, distances, and shapes of far heat-sourceobjects, the infrared cameras 16R, 16L are subject to an adjustmentprocess called an aiming process (which will be described later) whenthey are manufactured in the manufacturing plant or when they areinspected at regular intervals.

As shown in FIG. 2, the ECU 14 comprises an image input unit 40 forconverting analog infrared images obtained by the respective infraredcameras 16R, 16L into digital gray-scale images, a binarizer 42 forgenerating binary images from the gray-scale images based on a thresholdvalue, an image memory 44 for storing the binary images and thegray-scale images, an aiming mode execution unit 48 for storing cameraparameters produced as a result of the aiming process into a cameraparameter memory 46, a normal mode execution unit 50 for performing anormal image processing process while referring to sensors including thespeed sensor 22, etc. and the camera parameter memory 46, andcontrolling the HUD 18 and the speaker 20, and a mode selector 52 forselecting either an aiming mode or a normal mode at a time based on aninstruction transmitted from an external computer system through theconnector 32.

The aiming mode execution unit 48 has a manufacturing plant mode unit 70for performing the aiming process with an aiming target controlapparatus 100 (see FIG. 4) as the external computer system in themanufacturing plant in which the vehicle 12 is manufactured, and aservice mode unit 72 for performing the aiming process with a serviceaiming adjustment apparatus 120 (see FIG. 5) as the external computersystem in a service factory or the like. Either the manufacturing plantmode unit 70 or the service mode unit 72 is selected at a time based onan instruction from a corresponding one of the external computersystems.

The aiming mode execution unit 48 has a parameter input unit 74 forinputting certain parameters from the external computer system when theaiming process is initiated, an initializing unit 76 for making initialsettings required by the aiming process, a template matching unit 78 forperforming template matching on the gray-scale images stored in theimage memory 44, a luminance adjustment LUT setting unit 80 for settinga luminance adjustment LUT for adjusting the luminance of image signalsproduced by the infrared cameras 16R, 16L, a camera image distortioncorrecting unit 82 for correcting image distortions caused due toindividual differences as to focal lengths, pixel pitches, etc. betweenthe infrared cameras 16R, 16L, a camera mounting angle calculating unit84 for calculating respective mounting angles (a pan angle and a pitchangle) of the infrared cameras 16R, 16L, a camera image clippingcoordinate calculating unit 86 for calculating clipping coordinates usedas a reference for clipping processed ranges from images, and a parallaxoffset value calculating unit 88 for calculating a parallax offset valueas an error which is contained in the parallax between object imagesbecause the optical axes of the infrared cameras 16R, 16L are notparallel to each other.

The initializing unit 76 has a template setting unit 94 for selectingone of six templates TP1, TP2, TP3, TP4, TP5, TP6 (collectively alsoreferred to as “template TP”) that have been prepared depending on thedistance up to objects. The ECU 14 has a model memory 96 for storing, asa formula, a perspective transformation model for determining theposition of an object. The aiming mode execution unit 48 and the normalmode execution unit 50 calculate the position of an imaged object usingthe perspective transformation model stored in the model memory 96. Themodel memory 96 stores a short-distance model for objects at shortdistances and a long-distance model for objects at long distances.

FIG. 3 shows in block form an image extracting function of the normalmode execution unit 50. As shown in FIG. 3, the normal mode executionunit 50 has an object area setting unit 140 for clipping and setting anobject area from a binary image DR obtained by the infrared camera 16R(reference imaging unit) and stored in the image memory 44, a luminancesignal adjusting unit 142 for adjusting the luminance signal of an image(gray-scale image YR (IN)) obtained by the infrared camera 16R using theluminance adjustment LUT stored in the camera parameter memory 46, suchthat the luminance signal of the image (gray-scale image YR (IN))obtained by the infrared camera 16R is greater by a predetermined amountthan the luminance signal of an image (gray-scale image YL) obtained bythe infrared camera 16L (searching imaging unit), and a templatematching unit 144 (an image comparing unit, an image extracting unit)for extracting an image from the image (gray-scale image YL) obtained bythe infrared camera 16L according to correlative calculations, using anobject area in the image (gray-scale image YR (OUT)) obtained by theinfrared camera 16R and adjusted in luminance as a template.

The ECU 14 has a CPU (Central Processing Unit) as a main controller, aRAM (Random Access Memory) and a ROM (Read Only Memory) as a memorydevice, and other components. The above functions of the ECU 14 areimplemented in software when the CPU reads a program and executes theprogram in cooperation with the memory device.

As shown in FIG. 4, the aiming target control apparatus 100 haspositioning devices 102 for positioning the vehicle 12, a gate 104disposed in a given position in front of the infrared cameras 16R, 16Lon the vehicle 12 that is positioned by the positioning devices 102, anda main control device 106 for communicating with the ECU 14 through theconnector 32 and controlling the gate 104. The gate 104 has two verticalposts 108 horizontally spaced from each other by a distance which isslightly greater than the width of the vehicle 12, and a horizontallyelongate aiming target plate 110 having left and right ends movablysupported respectively by the posts 108. The aiming target plate 110 isvertically movable along the posts 108 by the main control device 106.The aiming target plate 110 supports thereon an array of eight aimingtargets 112 a through 112 h (collectively also referred to as “aimingtarget 112”) as heat sources that are successively arranged horizontallyfrom the left in the order named.

The four left aiming targets 112 a through 112 d are spaced atrelatively small intervals W and belong to a left target group 114. Thefour right aiming targets 112 e through 112 h are also spaced at theintervals W and belong to a right target group 116. The aiming target112 d on the right end of the left target group 114 and the aimingtarget 112 e on the left end of the right target group 116 are spacedfrom each other by a distance which is equal to the base length B (W<B).These aiming targets 112 d, 112 e are positioned just in front of theinfrared cameras 16L, 16R, respectively.

As shown in FIG. 5, the service aiming adjustment apparatus 120 haspositioning markers 122 for positioning the vehicle 12, a headlighttester 124 disposed in a given position in front of the infrared cameras16R, 16L on the vehicle 12 that is positioned based on the positioningmarkers 122, and a main control device 126 for communicating with theECU 14 through the connector 32. The headlight tester 124 is movablealong a rail 128 in directions parallel to the transverse direction ofthe vehicle 12 and has a lifter table 130 which is vertically movable.The lifter table 130 supports thereon a target plate 132 having threeaiming targets 134 a through 134 c (collectively also referred to as“aiming target 134”) as heat sources that are successively arrangedhorizontally. The aiming targets 134 a through 134 c are spaced at theintervals W (W<B). The aiming target 134 may be identical to orsubstantially the same as the aiming target 112 of the gate 104 shown inFIG. 4.

The aiming process to be performed on the night vision system 10 usingthe aiming target control apparatus 100 or the service aiming adjustmentapparatus 120 will be described below.

The aiming process includes a manufacturing plant aiming mode to beperformed using the aiming target control apparatus 100 and a serviceaiming mode to be performed using the service aiming adjustmentapparatus 120.

In the manufacturing plant aiming mode, the vehicle 12 is positioned bythe positioning devices 102, and the main control device 106 isconnected to the connector 32 of the vehicle 12. The main control device106 sends an instruction for performing the manufacturing plant aimingmode using the aiming target control apparatus 100 to the ECU 14. Theaiming targets 112 a through 112 h are positionally adjusted to the sameheight as the infrared cameras 16R, 16L depending on the type of thevehicle 12.

In the service aiming mode, the vehicle 12 is positioned with the wheelsaligned with the respective positioning markers 122, and the maincontrol device 126 is connected to the connector 32 of the vehicle 12.The main control device 126 sends an instruction for performing theservice aiming mode using the service aiming adjustment apparatus 120 tothe ECU 14. The aiming targets 134 a through 134 c are positionallyadjusted to a predetermined height.

FIGS. 6 through 10 show the aiming process that is mainly performed bythe aiming mode execution unit 48 of the ECU 14. The aiming process willbe described in detail below with reference to FIGS. 6 through 10.

In step S1 shown in FIG. 6, an analog stereographic infrared image isinput from the infrared cameras 16R, 16L to the image input unit 40. Theimage input unit 40 converts the analog stereographic infrared imageinto a digital gray-scale image in step S2. The gray-scale image isstored in the image memory 44. The gray-scale image is converted by thebinarizer 42 into a binary image, which is also stored in the imagememory 44.

In step S3, the mode selector 52 determines whether the aiming mode orthe normal mode is to be executed according to an instruction from themain control device 106 or 126. If the normal mode is to be executed,then control goes to step S5. If the aiming mode is to be executed, thencontrol goes to step S4.

In the normal mode in step S5, the normal mode execution unit 50operates to refer to the camera parameters stored in the cameraparameter memory 46, and controls the HUD 18 and the speaker 20 tosearch for an object and draw the driver's attention, as describedlater. Thereafter, control goes back to step S1.

In the aiming mode in step S4, the mode selector 52 determines which ofthe aiming target control apparatus 100 and the service aimingadjustment apparatus 120 is to be used. If it is judged that the aimingtarget control apparatus 100 is to be used, then control goes to step S6in order for the manufacturing plant mode unit 70 to perform themanufacturing plant aiming mode. If it is judged that the service aimingadjustment apparatus 120 is to be used, then control goes to step S30(see FIG. 8) in order for the service mode unit 72 to perform theservice aiming mode. The manufacturing plant aiming mode and the serviceaiming mode will successively be described below.

In the manufacturing plant aiming mode, a distance from the infraredcameras 16R, 16L to the aiming target plate 110 is set in step S6.

In step S7, the template setting unit 94 selects a reference templatecorresponding to the aiming target 112 from the templates TP1 throughTP6.

In step S8, in order to calculate the position of the aiming target 112using a perspective transformation model corresponding to the distanceup to the aiming target 112 disposed at a short distance, a focusingdistance (focal length) of the infrared cameras 16R, 16L which matchesthe perspective transformation model is set.

In step S9, a template matching process is performed based on thetemplate TP selected in step S7. Specifically, correlative calculationsare performed on a gray-scale image of the aiming target 112 obtained bythe infrared cameras 16R, 16L and the template TP, and coordinates of agray-scale image or a target for which the results of the correlativecalculations are minimum are calculated and stored in step S10.

In step S11, it is confirmed whether the number of acquired gray-scaleimages has reached a predetermined number N or not. If the number ofacquired gray-scale images has reached the predetermined number N, thencontrol goes to step S12. If the number of acquired gray-scale images issmaller than the predetermined number N, then control goes back to stepS1 to acquire another gray-scale image and calculate and store targetcoordinates.

In step S12, the N sets of target coordinates are averaged. If it isjudged that target coordinates are properly calculated in step S13, thencontrol goes to step S14 (see FIG. 7). If it is judged that targetcoordinates are not properly calculated in step S13, then control goesback to step S3.

In step S14, a luminance adjustment LUT is set. Specifically, in orderto reliably perform the template matching process based on correlativecalculations, the levels of luminance signals of the aiming target 112which are detected by the infrared cameras 16R, 16L are compared witheach other, for example, and a luminance adjustment LUT is set such thatthe luminance signal from the infrared camera 16R, which is used as areference for the correlative calculations, will be greater at all timesthan the luminance signal from the infrared camera 16L at each of theluminance levels. If it is judged that the process of setting aluminance adjustment LUT is properly performed in step S15, then controlgoes to step S16.

In step S16, an image distortion corrective value for correcting imagedistortions caused due to individual differences as to focal lengths,pixel pitches, etc. between the infrared cameras 16R, 16L is calculated.If it is judged that an image distortion corrective value is properlycalculated in step S17, then control goes to step S18.

In step S18, a pan angle and a pitch angle, which serve as mountingangles of the left and right cameras, i.e., the infrared cameras 16R,16L, are calculated. If it is judged that mounting angles of the leftand right cameras are properly calculated in step S19, then control goesto step S20.

In step S20, clipping coordinates for clipping image areas to beprocessed from the images obtained by the infrared cameras 16R, 16L arecalculated. If it is judged that clipping coordinates are properlycalculated in step S21, then control goes to step S22.

In step S22, a parallax offset value, which represents an errorcontained in the parallax between object images because the optical axesof the infrared cameras 16R, 16L are not parallel to each other, iscalculated. If it is judged that a parallax offset value is properlycalculated in step S23, then control goes to step S24.

In step S24, the luminance adjustment LUT, the image distortioncorrective value, the pan angle and the pitch angle, the clippingcoordinates, and the parallax offset value which are determinedrespectively in steps S14, S16, S18, S20, and S22 are stored in thecamera parameter memory 46. If these parameters are properly stored,then the manufacturing plant aiming mode is finished. At this time, theECU 14 sends a signal indicating that the manufacturing plant aimingmode is finished to the main control device 106. If the normal mode isto be subsequently executed, then a predetermined restarting process maybe performed. If the answers to the branching processes in steps S17,S19, S21, S23, and S25 are negative, then control goes back to step S3as when the answer to the branching process in step S13 is negative.

The service aiming mode will be described below. In the service aimingmode, steps S1 through S3 (see FIG. 6) are executed in the same manneras with the manufacturing plant aiming mode. Control then branches fromstep S4 to step S30 for the service mode unit 72 to perform a processingsequence shown in FIGS. 8 through 10.

In step S30 shown in FIG. 8, a distance from the infrared cameras 16R,16L to the target plate 132 is set. The distance from the infraredcameras 16R, 16L to the target plate 132 is determined by the positionof the aiming target 134 installed in a service factory where theservice aiming mode is performed. The distance is input from the maincontrol device 126 to the ECU 14.

In step S31, the height H (see FIG. 1) of the infrared cameras 16R, 16Lis confirmed and input.

In step S32, the template setting unit 94 selects one of the templatesTP1 through TP6 which corresponds to the distance to the aiming target134 set in step S30.

In step S33, a focusing distance matching the perspective transformationmodel which corresponds to the distance to the aiming target 134 is setin the same manner as with step S8.

In step S34, the position of the target plate 132 is confirmed.Specifically, in the service aiming mode, the target plate 132 is placedsuccessively in a central position PC, a left position PL, and a rightposition PR (see FIG. 5). When step S34 is executed for the first time,a signal for positional confirmation is sent to the main control device126 to place the target plate 132 in the central position PC. Inresponse to the signal, the main control device 126 displays a message“PLACE TARGET IN CENTRAL POSITION PC AND PRESS “Y” KEY” on the monitorscreen, for example. According to the message, the operator moves theheadlight tester 124 along the rail 128 either manually or with a givenactuator until the target plate 132 is placed in the central positionPC.

In step S35, control is branched depending on the position of the targetplate 132 at the time. If the target plate 132 is placed in the centralposition PC (in first through 30 cycles), then control goes to step S36.If the target plate 132 is placed in the left position PL (in 31stthrough 60th cycles), then control goes to step S41 (see FIG. 9). If thetarget plate 132 is placed in the right position PR (in 61st andsubsequent cycles), then control goes to step S46 (see FIG. 10).

In step S36, a template matching process is performed in the same manneras with step S9.

In step S37, target coordinates of the aiming target 134 are calculatedand stored in the same manner as with step S10.

In step S38, the number of acquired gray-scale images is confirmed inthe same manner as with step S11. If the number of acquired gray-scaleimages is 30 or more, then control goes to step S39. If the number ofacquired gray-scale images is smaller than 30, then control goes back tostep S1. In the second and subsequent cycles, steps S3 through S8 andsteps S30 through S35 are skipped.

In step S39, the target coordinates at the central position PC areaveraged in the same manner as with step S12. If it is judged thattarget coordinates are normally calculated in step S40, then controlgoes back to step S1. If target coordinates are not normally calculatedin step S40, then control goes back to step S3.

The target plate 132 is placed in the left position PL, and steps S41through S45 shown in FIG. 9 are similarly executed.

Then, the target plate 132 is placed in the right position PR, and stepsS46 through S50 shown in FIG. 10 are similarly executed.

If it is judged that target coordinates are normally calculated in finalstep S50, then control goes back to step S14 (see FIG. 7). Subsequently,the same process as the manufacturing plant aiming mode is performed,and camera parameters are stored in the camera parameter memory 46.

After the camera parameters are set in the aiming mode, the normal modeis carried out using the camera parameters.

An object searching process and, if necessary, an attention drawingprocess, in the normal mode will be described below with reference to aflowchart shown in FIG. 11.

An image of an area in front of the vehicle 12 is obtained as areference image by the infrared camera 16R, and an image of an area infront of the vehicle 12 is obtained as a searching image by the infraredcamera 16L. The obtained images are converted into respective gray-scaleimages YR (IN), YL, which are stored in the image memory 44, and thegray-scale image YR (IN) is converted by the binarizer 42 into a binaryimage DR, which is also stored in the image memory 44, in steps S101,S102.

The object area setting unit 140 (FIG. 3) of the normal mode executionunit 50 sets an object area 150 (see FIG. 12) including an object 148,e.g., a quadrangular area where values “1” of a binary image DR aresuccessively arranged in an x direction (horizontal direction) and a ydirection (vertical direction), using the binary image DR of a referenceimage 146 obtained by the infrared camera 16R in step S103. Then, theobject area setting unit 140 searches for an object 154 in a searchingimage 152, which corresponds to the object 148 in the reference image146, according to correlative calculations on the gray-scale images,calculates a parallax on the images between the objects 148, 154 thathave been searched for, and determines a distance from the vehicle 12 tothe object in step S104.

Prior to the correlative calculations, the luminance signal adjustingunit 142 adjusts the luminance signal of the reference image 146 or thesearching image 152 using the luminance adjustment LUT representative ofcamera parameters, such that the luminance signal of the reference image146 (gray-scale image YR (IN)) of the object area 150 obtained by theinfrared camera 16R is greater than the luminance signal of the image ofthe object area 156 of the searching image 152 (gray-scale image YL)obtained by the infrared camera 16L. Then, the template matching unit144 performs correlative calculations on the adjusted reference image146 (gray-scale image YR (OUT)) and the searching image 152 (gray-scaleimage YL) to search for the object 154 from the searching image 152.

The coordinates of the objects 148, 154 in the reference image 146 andthe searching image 152 are corrected by an image distortion coefficientwhich represents a camera parameter, and the parallax between theobjects 148, 154 that have been searched for is corrected highlyaccurately by a parallax offset value which represents a cameraparameter due to a relative pan angle. The distance from the vehicle 12to the actual object is calculated highly accurately.

Then, the two-dimensional coordinates of the object 148 in the referenceimage 146 are corrected by an absolute pan angle and an absolute pitchangle of the infrared cameras 16R, 16L obtained in step S18, and arelative position represented by three-dimensional coordinates of theobject 148 in the actual space, including the distance calculated instep S104, is calculated in step S105.

The three-dimensional coordinates of the object 148 in the actual spacein step S105 are repeatedly calculated at small time intervals tocalculate a moving vector of the object 148 in step S106. Using themoving vector, road structures and vehicles are removed from the object148 in step S107. Then, it is determined whether there is a pedestrianor not from the shape of the remaining object 148 in step S108.

If it is judged that there is a pedestrian in step S108, then thereference image obtained by the infrared camera 16R is displayed on theHUD 18, and the image of the pedestrian is enclosed by a highlightingframe in step S109. The speaker 20 is energized to draw the driver'sattention in step S110.

The reasons for setting a luminance adjustment LUT such that theluminance signal of the reference image 146 is greater than theluminance signal of the searching image 152 will be described below.

In FIG. 12, the luminance signal at coordinates (x, y) of the referenceimage 146 is indicated by YR (x, y), the luminance signal at coordinates(x, y) of the searching image 152 by YL (x, y), the numbers of pixels inthe x and y directions of the object area 150 by M, N, respectively, andthe distances by which the object area 156 is displaced from the objectarea 150 in the x and y directions by p, q, respectively. The templatematching unit 144 calculates a correlative function SAD (p, q) asfollows:

$\begin{matrix}{{{SAD}( {p,q} )} = {\sum\limits_{i = 0}^{M - 1}{\sum\limits_{j = 0}^{N - 1}{{{{YR}( {{x + i},{y + j}} )} - {{YL}( {{x + i + p},{y + j + q}} )}}}}}} & (1)\end{matrix}$

The object area 156 whose correlative function SAD (p, q) is minimum isextracted from the searching image 152.

Even if the correlative function SAD (p, q) is minimum, backgroundimages around the objects 148, 154 may possibly be detected as objectsin error due to different sensitivity characteristics and noise of theinfrared cameras 16R, 16L.

The luminance signal of the object 148 obtained by the infrared camera16R is indicated by a1, the luminance signal of the background aroundthe object 148 by b1, the luminance signal of the object 154 obtained bythe infrared camera 16L by a2, and the luminance signal of thebackground around the object 154 by b2. An evaluating value MAD1 for thecorrelative function SAD (p, q) if the objects 148, 154 match eachother, and an evaluating value MAD2 for the correlative function SAD (p,q) if the objects 148, 154 do not match each other are introduced asfollows:MAD1=|a1−a2|+|b1−b2|  (2)MAD2=|a1−b2|+|b1|b2|  (3)

By establishing a condition to satisfy the relationship:MAD2−MAD1=|a1−b2|−|a1−a2|>0  (4)

the object 154 can be searched for without detecting a background imagein error.

If a1>b1, a2>b2, and a1>a2, thenMAD2−MAD1=a2−b2>0  (5)

and the equation (4) is satisfied at all times (see FIG. 12).

If a2>a1 and a1>b2, then the equation (4) is satisfied (see FIG. 13)only whenMAD2−MAD1=2·a1−(a2+b2)>0  (6)

If a2>a1 and b2>a1, then the equation (4) is satisfied whenMAD2−MAD1=b2−a2>0  (7)

but the relationship b2>a2 never occurs (see FIG. 14).

Therefore, in order to be able to search for the object 154 at all timesaccording to correlative calculations without detecting a backgroundimage in error, the luminance signal of the image obtained by theinfrared camera 16R may be set so as to be greater at all times than theluminance signal of the same image obtained by the infrared camera 16L.

The luminance adjustment LUT setting unit 80 of the aiming modeexecution unit 48 sets a luminance adjustment LUT for adjusting theluminance signal depending on the level, such that a luminance signal YR(IN) produced when the infrared camera 16R obtains an image of theaiming targets 112 a through 112 h or 134 a through 134 c is greater atall times than the luminance signal YL produced when the infrared camera16L obtains an image of the aiming targets 112 a through 112 h or 134 athrough 134 c, for example. Then, the luminance adjustment LUT settingunit 80 stores the set luminance adjustment LUT in the camera parametermemory 46 (see FIG. 15).

The luminance signal adjusting unit 142 adjusts the luminance signal YR(IN) from the infrared camera 16R, which is to be adjusted in luminanceand which is read from the image memory 44, with the luminanceadjustment LUT that is representative of a camera parameter, and outputsthe adjusted luminance signal YR (IN) as a luminance signal YR (OUT) tothe template matching unit 144. The template matching unit 144 performscorrelative calculations on the adjusted luminance signal YR (OUT) fromthe infrared camera 16R and the luminance signal YL from the infraredcamera 16L to search for the object 154. Since the relationshiprepresented by the equation (4) is satisfied at all times, the object154 is reliably searched for without detecting a background image inerror.

The luminance adjustment LUT may be set by reading a gray scale havingstepwise luminance levels as an aiming target through the infraredcameras 16R, 16L, and setting the obtained luminance signal YR (IN) soas to be greater than the luminance signal YL at each of the luminancelevels. According to the luminance adjustment LUT, the luminance signalYL from the infrared camera 16L, which is to be adjusted in luminanceand which is read from the image memory 44, may be adjusted so as to besmaller than the luminance signal YR from the infrared camera 16R.Alternatively, an aiming target having a single luminance level may beread by the infrared cameras 16R, 16L, a single corrective value may beset which is capable of making the luminance signal YR (IN) greater thanthe luminance signal YL in the full luminance range, and the luminancesignal YR (IN) or YL may be adjusted using the single corrective value.

In FIG. 3, the infrared camera 16R is fixed as the reference imagingunit, and the luminance adjustment LUT is set using the image obtainedby the infrared camera 16R as the reference image. However, as shown inFIG. 16, the luminance signal YR from the infrared camera 16R and theluminance signal YL form the infrared camera 16L may be compared witheach other by a reference image selecting unit 160, and one of theinfrared cameras 16R, 16L which produces a greater luminance signal maybe selected as a reference imaging unit. The object area setting unit140 may set an object area 150 according to the binary image DR or DLfrom the selected reference imaging unit, and correlative calculationsmay be performed by the template matching unit 144 using the gray-scaleimage YR or YL from the reference imaging unit as a reference image.According to the embodiment shown in FIG. 16, the object 154 canreliably be extracted without the need for adjusting the luminance ofimages.

Although certain preferred embodiments of the present invention havebeen shown and described in detail, it should be understood that variouschanges and modifications may be made therein without departing from thescope of the appended claims.

1. An apparatus for extracting an image of one object by comparingimages obtained respectively by a plurality of imaging units,comprising: a luminance signal adjusting unit for using one of theimaging units as a reference imaging unit and another imaging unit thansaid reference imaging unit as a searching imaging unit, and adjusting aluminance signal from said reference imaging unit or said searchingimaging unit such that the luminance signal of a reference imageobtained by said reference imaging unit is greater by a predeterminedamount than the luminance signal of a searching image obtained by saidsearching imaging unit; an object area setting unit for setting anobject area including a particular object in said reference image; animage comparing unit for comparing an image of said object area and saidsearching image with each other; and an image extracting unit forextracting an image relating to said particular object from saidsearching image based on a comparison result from said image comparingunit; wherein said image comparing unit performs correlativecalculations on said images, and said image extracting unit extracts animage for which the results of the correlative calculations are minimumas said image relating to said particular object; wherein saidcorrelative calculations satisfy the equationMAD2−MAD1=a1−b2−a1−a2>0, Where MAD1 is an evaluating value for thecorrelative calculations if the particular objects in the reference andsearching images match each other, MAD2 is an evaluating value for thecorrelative calculations if the particular objects in the reference andsearching images do not match each other, a1 is the luminance signal ofthe particular object obtained by the reference imaging unit, b1 is theluminance signal of the background around the particular object by thereference imaging unit, a2 is the luminance signal of the particularobject obtained by the searching imaging unit, and b2 is the luminancesignal of the background around the object by the searching imagingunit.
 2. An apparatus according to claim 1, wherein said luminancesignal adjusting unit has an adjustment table for adjusting saidluminance signal of said reference image or said searching imageobtained by said reference imaging unit or said searching imaging unitdepending on the level of said luminance signal.
 3. An apparatusaccording to claim 1, wherein said luminance signal adjusting unit has asingle corrective value for adjusting said luminance signal of saidreference image or said searching image obtained by said referenceimaging unit or said searching imaging unit.
 4. An apparatus accordingto claim 1, wherein said imaging units are mounted on a vehicle.
 5. Anapparatus according to claim 1, wherein said imaging units compriseinfrared cameras for detecting infrared radiation radiated from saidparticular object.
 6. An apparatus according to claim 1, wherein saidluminance signal adjusting unit adjusts the luminance signal from saidreference imaging unit or said searching imaging unit, such that theluminance signal of the particular object which has a luminance higherthan that of a background in the reference image obtained by saidreference imaging unit is greater than the luminance signal of theparticular object in a searching image obtained by said searchingimaging unit, and wherein said image comparing unit performs correlativecalculations on the searching image and the object area set in thereference image, and said image extracting unit extracts an image forwhich the results of the correlative calculations are minimum as theimage relating to the particular object from the searching image, basedon a comparison result from said image comparing unit.
 7. An apparatusaccording to claim 1, wherein said luminance signal adjusting unitadjusts the luminance signal from said reference imaging unit or saidsearching imaging unit, such that the luminance signal of the particularobject which has a luminance higher than that of a background in thereference image obtained by said reference imaging unit is greater thanthe luminance signal of the particular object in a searching imageobtained by said searching imaging unit so that the image of theparticular object may be extracted from the searching image withoutrecognizing a background image in error.
 8. An apparatus according toclaim 1, further comprising an image distortion correcting unit whichcorrects image distortions due to individual differences between theimaging units.
 9. An apparatus for extracting an image of one object bycomparing images obtained respectively by a plurality of imaging units,comprising: an image selecting unit for selecting one of the imagesobtained by said imaging units; an object area setting unit for settingan object area including a particular object in said image selected bysaid image selecting unit; an image comparing unit for comparing animage of said object area and another image than said selected imagewith each other; and an image extracting unit for extracting an imagerelating to said particular object from said other image based on acomparison result from said image comparing unit; wherein said imageselecting unit selects an image having a greatest luminance signal fromthe images obtained by said imaging units; wherein said image comparingunit performs correlative calculations on said images, and said imageextracting unit extracts an image for which the results of thecorrelative calculations are minimum as said image relating to saidparticular object.
 10. An apparatus according to claim 9, wherein saidimaging units are mounted on a vehicle.
 11. A computer readable storagemedium storing instructions that, when executed by a computer, cause thecomputer to perform a method of extracting an image of one object bycomparing images obtained respectively by a plurality of imaging units,the method comprising the steps of: using one of said imaging units as areference imaging unit and another imaging unit than said referenceimaging unit as a searching imaging unit, and adjusting a luminancesignal from said reference imaging unit or said searching imaging unitsuch that the luminance signal of a reference image obtained by saidreference imaging unit is greater by a predetermined amount than theluminance signal of a searching image obtained by said searching imagingunit; setting an object area including a particular object in saidreference image; comparing an image of said object area and saidsearching image with each other; and extracting an image relating tosaid particular object from said searching image based on a comparisonresult of said comparing step; wherein said image comparing unitperforms correlative calculations on said images, and said imageextracting unit extracts an image for which the results of thecorrelative calculations are minimum as said image relating to saidparticular object; wherein said correlative calculations satisfy theequationMAD2−MAD1=a1−b2−a1−a2>0. where MAD1 is an evaluating value for thecorrelative calculations if the particular objects in the reference andsearching images match each other MAD2 is an evaluating value for thecorrelative calculations if the particular objects in the reference andsearching images do not match each other, a1 is the luminance signal ofthe particular object obtained by the reference imaging unit, b1 is theluminance signal of the background around the particular object by thereference imaging unit, a2 is the luminance signal of the particularobject obtained by the searching imaging unit, and b2 is the luminancesignal of the background around the object by the searching imagingunit.
 12. A method according to claim 11, wherein said step of comparingcomprises the step of performing correlative calculations on said imageof said object area and said searching image, and said step ofextracting comprises the step of extracting an image for which theresults of the correlative calculations are minimum as said imagerelating to said particular object.
 13. A computer readable storagemedium storing instructions that, when executed by a computer, cause thecomputer to perform a method of extracting an image of one object bycomparing images obtained respectively by a plurality of imaging units,the method comprising the steps of: selecting an image having a greatestluminance signal from the images obtained by said imaging units; settingan object area including a particular object in said selected image;comparing an image of said object area and another image than saidselected image with each other; and extracting an image relating to saidparticular object from said other image based on a comparison result ofsaid step of comparing; wherein said step of comparing comprises thestep of performing correlative calculations on said image of said objectarea and said other image, and said step of extracting comprises thestep of extracting an image for which the results of the correlativecalculations are minimum as said image relating to said particularobject.